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Abstract 

Pyramids  are  data  structures  used  to  store  and  process  images  at 
multiple  levels  of  resolution.  The  bottom  level  of  a  pyramid  is 
used  to  represent  data  at  a  fine  level  of  resolution,  while  higher 
levels  of  the  pyramid  are  used  for  data  stored  at  coarser  levels 
of  resolution.  For  example,  in  the  Gaussian  pyramid  data  struc- 
ture, each  successive  level  is  obtained  by  local  averaging  and 
subsampling  of  the  immediately  lower  level  in  the  pyramid.  In 
nearly  all  pyramid  implementations  to  date,  the  size  reduction  in 
each  dimension  between  levels  of  the  pyramid  is  a  constant  fac- 
tor of  two. 

This  paper  describes  a  scheme  that  permits  construction  of  py- 
ramids with  arbitrary  size  reductions  between  levels.  The  reduc- 
tion factors  can  be  different  in  each  dimension,  and  differ 
between  levels,  to  adapt  to  a  given  appHcation.  The  user  can 
thus  specify  a  sequence  of  decreasing  rectangular  image  sizes, 
and  construct  pyramids  conforming  to  those  sizes.  Further,  the 
reduction  factors  can  be  made  adaptive  to  region  properties,  ena- 
bling smooth  regions  to  be  reduced  more  than  "busy"  regions. 


1.  Pyramids 

Pyramid  data  structures  have  proven  useful  in  the  development  of  image  pro- 
cessing methods  dealing  with  image  features  at  varying  scales  of  resolution.  A  sur- 
vey on  pyramid  structures  may  be  found  in  [1].  Typical  image  pyramids  are  formed 
from  rectangular  grid  arrays  whose  sides  have  lengths  that  are  powers  of  two  in 
extent.  Thus  the  base  level  might  be  512  by  512,  the  next  level  up  would  then  be 
256  by  256,  and  each  successive  level  reduces  each  dimension  by  a  factor  of  two.  In 
the  Gaussian  pyramid  data  structure,  the  reduction  from  one  level  to  the  next  is 
accomplished  by  blurring  the  lower  level  (by  means  of  convolution  with  a  non- 
negative  kernel)  followed  by  a  subsampling  of  every  other  pixel  on  every  other  row. 
In  the  Burt  form  of  the  Gaussian  pyramid  [2],  each  level  has  size  of  the  form  2"  +  l 
by  2"+  1,  so  that  in  the  subsampHng  operation  left  and  right  edge  pixels  are  included 
on  every  second  row,  and  that  both  the  top  and  bottom  rows  are  sampled.  Nonethe- 
less, the  sampling  rate  is  still  two,  and  we  say  that  the  size  ratio  between  levels  is 
two  in  each  dimension. 

By  subsampling  every  other  pixel  on  every  row  and  offsetting  the  samples  by 
one  pixel  on  successive  rows,  it  is  possible  to  achieve  an  effective  sampling  ratio  of 
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\/2  in  each  dimension  [3].  In  this  pyramid  scheme,  every  other  level  is  in  essence 
located  on  a  45°  grid,  and  care  must  be  exercised  when  designing  kernels  and  other 
local  operators  on  these  levels.  Nonetheless,  the  reduced  reduction  factor  between 
levels  results  in  a  better  chance  of  capturing  a  salient  feature  at  an  appropriate  scale 
of  resolution,  at  the  cost  of  doubling  the  number  of  levels. 

In  this  paper,  we  describe  a  method  for  constructing  pyramids  of  arbitrary  size 
at  each  level,  so  that  the  resolution  can  be  reduced  as  needed,  and  not  only  by  fixed 
ratios.  Further,  the  reduction  does  not  have  to  be  uniform  over  the  whole  image, 
and  can  vary  based  on  local  image  properties. 

The  basic  step  in  the  construction  of  the  proposed  pyramid  scheme  is  a  spatial 
resampling  technique  used  in  graphics  [4,5],  related  to  anti-aliasing.  We  will  first 
describe  the  resampling  idea  in  one  dimension.  The  transformation  can  be  applied 
to  rectangular  grids  by  first  sampling  the  rows,  and  then  sampling  the  columns  of 
the  result.  We  will  then  present  a  formulation  of  the  sampling  idea  that  permits  the 
construction  of  pyramids  using  arbitrary  placements  of  pixels  (such  as  hexagonal 
grids).  The  central  property  of  the  sampling  method  is  that  each  pixel  contributes 
fully  to  the  output  samples,  thereby  minimizing  sampling  artifacts. 

As  an  introduction  to  the  resampling  methods  to  be  defined  in  later  sections, 
consider  the  original  sample  pomts  as  "producers,"  and  the  new  sample  points  that 
form  the  resampled  data  as  the  "consumers."  The  producers  and  consumers  have 
associated  positions,  corresponding  to  the  sample  point  locations  in  the  domain. 
The  amount  "produced"  by  each  producer  is  pre-specified,  as  is  the  consumption 
level  of  each  consumer.  The  consumers  obtain  their  products  from  a  linear  sum  of 
the  producers,  taking  contributions  of  fixed  amounts,  such  that  the  total  of  all  con- 
tributions by  a  given  producer  is  the  total  amount  produced  at  that  site.  The 
"error"  of  a  sampling  can  be  defined  as  the  sum  over  all  producer-consumer  pairs 
of  the  distance  between  the  corresponding  points  weighted  by  the  contribution  of  the 
producer  to  the  consumer.  The  error  is  minimized  when  the  consumers  use  the 
closest  suppliers  to  compute  their  values.  The  methods  described  below  have  cer- 
tain optimality  properties  with  respect  to  these  measures,  but  we  will  not  pursue  the 
variational  derivation  any  further. 


2.  One-Dimensional  Resampling 

We  first  consider  the  problem  of  resampling  a  vector  (vq,  •  •  •  .vat-i)  of  A^  pix- 
els to  a  vector  (wo,  •  •  •  ,y^M-\)  of  ^  pixels.    We  treat  both  cases  M  <  N  and 

M  ^  N. 

2.1.  Uniform  Resampling 

We  use  the  formula 

r(<+i)/pi-i 


j=  Li/pJ 


where 
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min  {  1,  p,  (  pO  +  1)  -  /)  }  if    U/pJ  ^  ;  <  i'P 
^  min{  1,  p,  (  (z  +  1)  -  p;)  }  if  z7p  <  ;  <    r(/  +  l)/pl 


and 


P  = 


To  interpret  this  formula,  first  assume  that  A/  <  A^,  so  that  p  is  small.  Then 
nearly  all  r,-^  for  j  in  the  appropriate  range  will  have  the  value  p,  with  the  exception 
of  the  two  extreme  n/s. 

More  generally,  we  can  regard  the  output  range  [0,A/]  split  up  into  M  subinter- 
vals  [/,  /  +  1],  with  /  =  0,1,  •  •  ■  ,  M— 1.  The  same  range  is  also  split  into  A^  subin- 
tervals  [pj,  p'(y  +  l)],  with  j  =  0,1,  •  •  •  ,  /V  — 1.  The  coefficient  rij  is  the  total 
length  of  the  portion  of  the  interval  [p;,  p(y  +  l)]  that  intersects  with  the  interval 
[/.  /  +  !]. 

For  the  situation  with  N  ~  4  and  M  =  3,  to  resample  (vo,vi,V2,V3)  with 
(wo, w  1,^2)  we  have  the  formulas 

3        ,     1 
Wo  =  -jvo  +  —VI 

wi  =  —vi  +  —V2 

W2  =   ■7V2  +  "TV3  ■ 

as  shown  in  Figure  la.    Figure  lb  shows  how  the  coefficients  can  be  computed  from 
the  overlapping  intervals. 

Properties  of  this  sampling  method  include: 

N-l 

•  The  total  contribution  of  any  given  Vj  to  all  w's  is  p;  i.e.,   "^  rij  =  p. 

1=0 

M-l 

•  The  sum  of  all  contributions  to  any  given  w,-  is  one;  i.e.,   ^  r^  =  1. 

;  =  0 


VQ  Vj  V2  V3 

Figure  la.  Uniform  resampling  weights  for  a  4-vector  to  a  3-vector. 
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H-O 
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Vl 
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Figure  lb.    Method  for  computing  the  uniform  resampling  weights  for  a  4-vector  to 
a  3-vector,  using  the  overlap  of  the  intervals  at  different  resolutions. 


2.2.  Weighted  Resampling 

In  the  uniform  sampling  given  in  the  previbus  section,  all  input  pixels  make  the 
same  contribution  p  to  output  pixels.  We  now  extend  the  resampling  to  permit  each 
input  pixel  to  have  an  adaptable  "forward  weight,"  so  that  pixel  ;,  with  value  vj, 
contributes  a  total  of  \i.j  to  the  w's.  These  weight  should  satisfy  the  normalization 
condition 

N-l 
j  =  0 

reflecting  the  desire  to  have  each  of  the  M  output  pixels  to  receive  unity  in  contribu- 
tions from  the  v's.  When  \i.j  =  M IN  for  all  ;,  we  have  the  uniform  sampling  of  the 
last  section. 

We  use  the  same  principle  as  in  the  previous  section  to  develop  a  linear  resam- 
pling formula 

j  =  0 

but  now  Kij  represents  the  length  of  that  portion  of  the  interval 


that 


h  Jij, 


Jl:=0         it  =  0 

intersects  the  interval  [i,  i  +  l].  Note  that  the  input  interval  number  ;  has  lengt 
and  that  the  input  intervals  subdivide  the  output  range  [0,M]. 

We  omit  the  precise  formulas  for  the  fy's,  but  depict  the  situation  for  N  =  4, 
M  =  3,  and  M-o  =  JJi-i  =  1.  i'^2  -  \^3  —  0-5,  in  Figure  2.  In  this  case  we  obtain  the 
formulas 


wo 


=  vo 


W2   = 


Vl 
1 


2V2+yV3 


We  can  formulate  the  general  weighted  resampling  formulas  by  giving  an  inter- 
polation formula  and  a  sampling  formula.  Specifically,  given  a  vector 
(vo.vi,  ■  •  •  ,VN-i)   and  the   weights   (jxo,  •  •  '  ,l^N-l),  we  form   an   interpolation 
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vo 


VI 


Figure  2a.    Pyramid  representations  of  the  weighted  sampling  with  weights  as  in  the 
text. 


M-O 

wi 

M'2 

VO 

"1 

V2 

V3 

H-O  =  1  Jii  =  1  V-2  —  0-5       ^-3  T  0-5 

Figure  2b.    Weighted  resampling  of  a  4-vector  to  a  3-vector. 


function 


where 


1  =  0 


4>,(;c)  = 


i-l  I 

1   if    S  ^L-t  ^  J^  <  2  M-t 

*  =  0  Jt  =  0 

^  0  otherwise    . 


The  samples  (wq,  •  •  •  ,yJM-\)  are  then  obtained  from  the  sampling  formula 

w,-  =  I  f{x)^i{x)dx  , 
where 


[p  otherwise   . 


This  formulation  suggests  a  generalization,  to  which  we  return  in  the  next  section. 
For  the  moment,  we  note  that  the  fy's  satisfy 


AT-l 


The  total  contribution  of  any  given  v^  to  all  w's  is  \Ij;  i.e.,   "^  rij  =  \lj. 

i=0 
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M-1 


•       The  sum  of  all  contributions  to  any  given  w,  is  one;   i.e.,   ^  r^  —  1. 

j  =  0 

3.  Pyramid  Construction 

An  input  image  of  size  M  by  N  can  be  resampled  to  any  desired  output  size  K 
by  L  using  the  uniform  resampling  method  of  Section  1.1,  applying  first  a  resam- 
pling of  the  rows  (which  are  ^/-vectors)  to  have  lengths  L,  and  then  resampling  the 
resulting  L  columns  (each  of  which  are  M-vectors)  to  have  lengths  K.  The  resuh  is 
the  same  if  the  columns  are  first  resampled,  followed  by  a  resampling  along  the 
resulting  rows.  A  complete  pyramid  structure  can  be  built  by  specifying  the  desired 
sizes  of  all  levels  above  the  base  level. 

In  Figure  3,  we  present  an  example  of  a  three-level  one-dimensional  pyramid. 
Note  that  the  values  in  higher  levels  are  weighted  averages  of  bottom-level  pixels, 
and  that  the  supports  of  the  linear  weighting  functions  can  overlap  in  lower  levels. 
In  the  immediately  preceding  level  in  a  one-dimensional  pyramid,  with  resampling 
as  given  in  Section  1.1  or  1.2,  support  functions  for  adjacent  pixels  can  overlap  in  at 
most  one  pixel.  However,  the  supports  from  lower  levels  can  have  arbitrarily  large 
overlaps,  depending  on  the  structure  of-ihe  intermediate  levels.  Burt's  overlapping 
pyramids  [2]  also  exhibit  overlapping  supports,  although  in  the  one-dimensional  ver- 
sion of  his  scheme  (using  five-tap  filters)"/ adjacent  pixels  in  one  level  share  supports 
from  three  pixels  in  the  immediately  preceding  level  (see  Figure  4).  Successive  lev- 
els have  larger  overlaps  in  the  base  level.  However,  resampling  using  either  the 
uniform  resampling  or  weighted  resampling  of  Section  1.1  or  1.2  yields  weights 
determined  by  the  sizes  of  the  levels,  and  are  not  adjustable  in  the  same  way  that 
the  taps  in  the  Burt  pyramid  can  be  modified. 


9 

1 1  \ 


/   I  \ 


3  1       6/^6       13 
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Figure  3.  A  comparison  of  a  4-3-2  one  dimensional  pyramid  with  a  4-2  pyramid 
structure.  The  leftmost  pyramid  shows  the  weights  in  a  4-3-2  structure,  while  the 
middle  pyramid  shows  the  effective  weights  from  the  bottom  level  to  the  top.  The 
rightmost  pyramid  gives  the  weights  for  a  uniform  4-2  resampling. 
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Figure  4.    The  Burt  5-tap  pyramid. 


Using  the  interpolation  formula  and  sampling  method  formulation  of  resam- 
pling, however,  we  can  gencraiizc  the  uniform  and  weighted  resampling  methods  to 
permit  larger  overlaps  in  the  support  functions  of  the  resampling  kernels.  Rather 
than  defining  (t>,(x)'s  and  i}j,(x)'a  as  in  Jifcction  2.2,  we  instead  simply  assume  that 
the  interpolation  kernels  <}>,•(;.•)  f'or.a  'i  ;>uiiition  of  unity  of  the  domain,  and  that  the 
sampling  kernels  v]/,(x)  all  have  unit  mass.    By  this,  we  mean  that 

S  <t>/U)  ^  1 

1=0 


and 


f}\ii(x)dx  =  1,     for  1  =  0,  •  •  •  ,  M. 


In  this  case,  resampling  is  still  done  by  the  formula  w/  =  '^rijVj,  but  now  the  r^y's 
are  given  by 

fij  =  J^i(x)^j(x)dx. 

In  all  cases,  the  total  of  all  contributions  to  a  given  pixel  from  all  pixels  at  some 
given  lower  level  will  be  unity. 

The  pyramid  resampling  scheme  can  be  used  for  building  successively  smaller 
size  levels  (as  in  a  Gaussian  pyramid),  or  for  expanding  a  level  to  a  larger  image. 
By  combining  a  subsampling  operation  for  contraction  with  expansion  operations, 
the  resamplings  may  be  used  for  building  analogs  to  the  Laplacian  pyramid  data 
structure.   The  construction,  analogous  to  Burt's  formulation,  proceeds  as  follows. 

We  specify  a  sequence  of  decreasing  sizes  (A^,-  by  M,)  for  levels 
/,  I  =  0,  1,  •  •  •  ,  €  of  a  pyramid  structure.  Let  Gq  be  the  Nq  by  Mq  initial  image. 
We  construct  the  Laplacian  pyramid  as  a  sequence  of  images  L,-  of  size  TV,  by  M,-, 
I  =  0,  1,  •  •  •  €.  Recursively,  for  z  <  €,  G,  +  i  is  obtained  from  G,-  by  resampling 
from  an  A^,-  by  M,-  image  to  an  /V,  +  i  by  A/,  +  i  image: 

Gi  +  l  =  T^(Ni  X  Mi)  -  {Ni+i  X  Mi+i)    (Gi)  . 

where  7Z  is  the  resampling  operator.  We  assume,  for  the  time  being,  that  uniform 
resampling  will  be  used.  Then  L,-  is  defined  as  the  difference  between  G,-  and  an 
expansion  resampling  of  G,  +  i: 
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^i  —  ^i  ~  '^(Ni  +  i  X  Mi  +  i)  ~(Ni  X  Mi)     (G,  +  i)   . 

Finally,  at  the  top  level,  we  define  L(  =  Gf. 

As  with  the  Burt  Laplacian  pyramid,  the  original  image  Go  can  be  recon- 
structed exactly  from  the  L,'s  using  the  (obvious)  formulas 

Ge  =  Li 

G,-l   =  L,_i   +   TZ(Mi  X  Mi)  ~  (Ni-i  X  Mi-i)     (G|)   • 

Each  level  L,-  represents  an  approximate  band-pass  filtered  version  of  the  original 
image,  where  the  widths  of  the  bands  are  determined  by  the  choices  of  the  relative 
sizes  of  the  levels.  Figure  5  shows  such  a  custom-buih  Laplacian  pyramid  structure. 

Suppose  that  we  wish  to  use  weighted  resampling  in  the  construction  of  the 
Laplacian  pyramid.  It  is  then  desirable  to  invert  (in  some  sense)  the  weighted 
resampling  that  was  used  in  the  contraction  from  level  G,-  to  G,  +  i  for  the  expansion 
operation  in  the  construction  of  L,-.  The  same  weighted  expansion  should  then  be 
used  in  the  reconstruction  of  G,  using  )the  levels- L,- + 1 ,  •  •  •  ,  L^.  A  weighted  expan- 
sion resampling  should  be  used  (even,  though  it  is  not  absolutely  required  for  exact 
reconstruction)  so  that  the  levels  Lf  m^kHauf-theif  approximate  bandpass  charac- 
teristics.  What  weights  should  be  used?     ;•,        ,  ^ 

Assume  that  the  data  (vq,  •  •  •  ,vn-i)  has  been  resampled  using  weights 
(p-o,  ■  •  •  ,\i'N-l)  to  obtain  the  A/-vector  (wq,  •  •  •  .wjiz-i).  Recall  that  the  resam- 
pling is  linear: 

N-l_ 
j  =  0 

The  matrix  coefficients  r^  represent  the  contribution  fractions  of  coefficient  vj  to 
coefficient  w,.  We  now  define  the  "backward  weights"  P,-,  i  =  0,  •  •  •  ,  M—l, 
according  to 

j  =  0  i^j 

M-l 

It  is  easy  to  see  that   2  Pi  ~  ^'  ^°  ^^^^  ^^^  P''^  ^^^  serve  as  weights  for  resampling 

i=0 
the  M-vector  (wq,  •  •  •  ,wm-i)  to  an  iV-vector  (vq',  •  •  •  ,VAr_i').  Each  p,- 
represents  the  sum  of  proportions  of  v/s  that  contributed  to  w,-.  Accordingly,  the 
expansion  using  weights  (Po,  •  •  •  .Pw  -  l)  leads  to  a  vector  (vq',  •  •  •  vn-i')  which 
best  approximates  (vo,  •  •  ■  ,vn  -  i),  in  the  sense  that  there  is  no  horizontal  skew- 
ing. 

For  example,  uniform  resampling  of  an  A^-vector  to  an  M-vector  corresponds  to 
weighted  sampling  where  the  weights  are  all  equal  to  p  =  M/N.  The  backward 
weights  are  then  also  all  equal,  and  have  value  p"^  =  N/M.  Thus  uniform  resam- 
pling for  contraction  is  paired  with  uniform  resampling  for  expansion. 

As  another  example,  we  showed  in  Figure  2  the  weighted  sampling  of  a  4- 
vector  to  a  3-vector  with  weights  (1,  1,  1/2,  1/2).  The  resulting  backward  weights 
are  (1,  1,  2),  so  that  the  weighted  backward  resampling  becomes 
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Figare  5a 
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Figure  5b 

Figure  5.  A  custom-built  two-dimensional  Laplacian  pyramid  with  level  sizes  as 
marked.  Levels  are  enlarged  to  full  size  for  visibility.  Figure  5a  show  the  Gaussian 
pyramid,  while  the  Figure  5b  contains  the  levels  of  the  Laplacian  pyramid.  The  ab- 
solute value  of  the  data  is  displayed,  with  larger  values  shown  more  darkly. 
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as  depicted  by  Figures  6a  and  6b.  The  total  effect  of  the  contraction  followed  by  the 
expansion  is  that  the  higher  weighted  nodes  were  exactly  reconstructed,  while  the 
lower  weighted  nodes  were  blurred. 


4.  Adaptive  Resampling 

Weighted  resampling,  as  introduced  in  Section  1.2,  has  the  advantage  that 
regions  with  "interesting"  activity  can  be  resampled  more  finely  than  "uninterest- 
ing" regions.  This  can  be  accomplished  (in  one  dimension)  by  giving  interesting 
pixels  larger  weights.  In  this  Section,  we  suggest  an  interest  operator  for  obtaining 
the  resampling  weights,  and  also  discuss  extensions  to  two  dimensions  and  irregular 
tessellation  grids. 

4.1.   One-dimensional  Adaptive  Pyramid 

We  suggest  an  interest  operator  based  on  the  local  "busyness"  of  the  data.  It 
has  been  observed  that  in  human  perception  a  line  with  higher  "busyness"  seems 
longer  than  a  straight  line  segmsnt  [6],  as  in  Figure  7.  Here,  we  will  use  a 
smoothed  absolute  value  of  the  Laplacian  of  the  data  to  measure  "busyness."  A 
similar  operator  has  been  suggested  for  representation  of  intensity  information  by 
retinal  receptive  fields  [7]. 


Figure  6a.  Backward  weights  associated  with  the  weighted  resampling  of  Figure  2. 
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P2 

=  2 

>*'o 

H-l 

W2 

vo' 

vi' 

V2' 

V3' 

Fignre  6b.   Calculation  of  the  resampling  weights  using  the  backward  weights. 
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Figure  7.    An  illusory  distortion  in  which  the  divided  space  appears  longer  than  the 
undivided  space  (Oppel-Dundt  illusion). 


Specifically,  given  a  signal  t'c.^'l.  *  *  "  .va'-i,  we  compute  the  absolute  Lapla- 
cian  values: 

^i  =     I  v,-i  -  2v,-  +  v,+i  i  ,     I  =  1,  •  •  •  ,  A^-2, 

and  then  smooth  the  results  by  any  reasonable  local  averaging  operator,  extrapolate 
to  include  the  endpoints  /  =  0  and  i  —  N—l,  and  normalize  to  obtain  the  values 
|x,-,  /  =  0,  •  •  •  ,N  —  l.  One  way  to  accomplish  the  smoothing  is  to  define  an  itera- 
tive weighted  smoothing  as  follows: 

boj  =  lj,  l<;<A^-2. 

bo,N-l   =  ^N-l     , 

and  recursively  set 


4 
4' 


fcf+1,0  -  T^'.o  "''  T^'- 1  ' 


bt+i,N-i  =  -^bt,N-2  +  -^bt^N-i  ■ 
Finally,  set 

N-bT,i 

\^i=  TTTi 

j  =  0 

for  some  fixed  integer  7  >  0  representing  the  amount  of  smoothing.    The  ^i,'s  are 
used  as  weights  for  resampling  the  v,'s. 

An  adaptive  custom-made  one-dimensional  Laplacian  pyramid  can  therefore  be 
easily  constructed.  The  sizes  of  the  levels  are  pre-specified.  The  signal  data  is 
given,    and    weights    are    obtained    from    the   busyness    measures    of   that    data. 
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Resampling  using  these  weights  results  in  the  data  immediately  above  the  base 
level.  Recomputing  new  busyness  measures  gives  new  weights  that  can  be  used  for 
the  next  level  resampling;  continuing  in  this  fashion  yields  a  Gaussian  pyramid  with 
adaptive  weights.  To  obtain  the  Laplacian  pyramid,  each  level  above  the  base  must 
be  expanded  to  the  size  of  the  previous  level.  This  must  be  done  using  the  back- 
ward weights,  as  described  in  Section  3.  The  resulting  expanded  levels  are  differ- 
enced with  the  Gaussian  pyramid  data  at  that  level  to  obtain  the  Laplacian  data. 
Reconstruction  from  the  Laplacian  pyramid  structure  is  possible,  but  requires  that 
expansion  resampling  using  the  appropriate  backward  weights  at  each  level  be  used. 
This  means  that  the  complete  adaptive  Laplacian  pyramid  data  structure  consists  of 
the  Laplacian  data  at  each  level  together  with  the  backward  weights  needed  for  the 
expansion  resampling  at  each  level  above  the  base. 

4.2.  Two-dimensional  Adaptive  P^^ramids 

The  two-dimensional  case  involves  technical  difficulties  not  present  in  the  one- 
dimensional  case.  The  problems  avizt  due  to  the  fact  that  in  one  dimension,  weights 
can  be  converted  to  interval  lengths  to  decompose  the  domain,  whereas  in  two 
dimensions  it  is  not  clear  how  to  chan^^e  the  size  of  a  rectangular  region  in  relation 
to  a  weight  and  still  have  a  complete  decomposition  of  the  domain. 

One  solution  is  to  resample  "n  each  dimension  separately.  However,  if  each 
row  or  column  is  processed  independently,  then  the  image  will  lose  its  structure  in 
the  sense  that  neither  connectivity  or  convexity  of  shapes  will  be  preserved.  Since 
the  weights  on  one  row  may  be  independent  and  different  from  the  weights  on  a 
neighboring  row,  nearby  pixels  may  give  principle  contributions  to  distant  pixels  in 
the  next  level.  Since  this  behavior  is  typically  unacceptable,  we  suggest  a  scheme 
where  all  rows  and  all  columns  use  the  same  weights.  In  this  case,  the  row  weights 
will  be  the  average  row  vector  of  a  busyness  matrix,  and  the  column  weights  will  be 
the  average  column  vector  of  the  same  matrix. 

Specifically,  we  compute  a  "busyness  measure"  bij  at  each  pixel  (i,j)  in  the  A^ 
by  M  image.  Suppose  that  we  wish  to  resample  to  an  N '  by  M'  image.  Then  the 
row  weights,  used  to  resample  an  M-vector  to  an  M' -vector,  are  given  by 

N-l 

M'-^bij 


y^j 


1  =  0 


1=0     t=0 


and  the  column  weights,  used  to  resample  each  column  yV-vector  to  an  //'-vector, 
are  given  by 

N'-^bij 

V,  = i^ — 

'  N-l    M-\ 

k=Q     7=0 

The  same  weights  (|xo,  ■  •  •  ,\lm-])  are  used  for  resampling  every  row,  and  the 
weights  (vo,  •  •  •  ,vn-\)  are  used  for  resampling  the  columns. 
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Once  again,  a  possible  method  for  computing  "busyness  values"  is  to  smooth 
absolute  values  of  the  Laplacian  of  the  image  data. 

As  a  trivial  example,  suppose  that  we  wish  to  resample  a  4  by  4  image  to  a  2 
by  2  image,  and  that  the  busyness  matrix  has  form 

0  0  0  0 

0  0  2  2 

0  0  2  2 

0  0  2  2 

Then  the  row  weights  are  given  by  (0,  0,  1,  1),  and  the  column  weights  are 
(0,  2/3,  2/3,  2/3).  The  row  resampling  will  then  work  as  shown  in  Figure  8a,  and 
the  column  resampling  is  as  shown  in  Figure  8b.  Note  that  the  result  is  that  the  3  by 
2  bottom  right  subimage  will  be  resampled  to  a  2  by  2  image,  and  that  the  other  pix- 
els will  be  ignored. 

A  contrasting  example  arises, if  the  busyness  matrix  has  the  form 

2  2  0  0 

2  2  0  0  •  ■    ^v  '  ' 

0  0  2  2  '"'""'  ' 

0  0  2  2  .  :i    ■ 

In  this  case,  the  row  and  column  weights  are  both  (1/2,  1/2,  1/2,  1/2),  and  the 
resampling  for  both  the  rows  and  columns  will  be  uniform.  The  non-busy  quadrants 
(the  upper  left  and  lower  right  portions)  of  the  4  by  4  image  can  not  be  contracted 
without  distorting  shapes  in  the  remaining  portions  of  the  image. 

Figure  9  shows  an  adaptive  custom-made  pyramid  of  an  image.  Note  how  the 
interesting  regions  in  the  image  become  stretched  into  larger  windows  at  the  higher 
levels. 

5.  Irregular  Tessellations 

The  non-adaptive  versions  of  the  resampling  methods  given  in  previous  sec- 
tions easily  extend  to  grid  arrays  other  than  rectangular  lattices.    Hexagonal  grids 


vo  Vl  V2  V3 


Figure  8a.    Row  resampling  using  adaptive  weights  of  the  simple  example. 
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Figure  8b.    Column  resampling  using  adaptive  weights  of  the  simple  example. 


Figure  9a 
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Figure  9b 

Figure  9.  A  six  level  adaptive  pyramid,  with  level  sizes  as  marked.  Figure  9a 
shows  the  levels  of  the  Gaussian  adaptive  pyramid,  and  the  Figure  9b  shows  the  La- 
placian  adaptive  pyramid. 


are  often  suggested  as  a  useful  arrangement  for  image  processing.  There  are  hexag- 
onal grids  where  the  cell  sizes  expand  with  increasing  distance  from  a  central  cell 
[8],  and  such  grids  may  have  a  basis  in  human  retinal  cell  distributions  [9].  We 
might  also  imagine  a  random  placement  of  sample  points,  with  local  density  of  sam- 
pling points  prescribed  by  some  rule  (perhaps  adaptive  to  the  image  data).  We  will 
first  discuss  the  general  case,  and  then  focus  on  a  case  where  sample  point  locations 
are  described  on  a  polar  grid. 
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5.1.  General  case 

In  all  cases,  the  locations  of  the  sampling  points  are  associated  with  either  a 
regular  tessellation  or  irregular  tessellation  (a  decomposition)  of  the  image  domain 
into  disjoint  cells.  Each  sample  v,-  represents  its  corresponding  cell  C,-,  and  the 
union  of  all  such  cells  covers  the  domain.  If  a  tessellation  is  not  provided  as  part  of 
the  sampling  locations,  then  one  can  be  provided  by  constructing  the  Voronoi 
decomposition  associated  with  the  points  [10]. 

Given  one  such  decomposition  of  the  domain,  say  {C,}fLV»  and  another 
(perhaps  coarser)  decomposition,  say  {D,}/lo^,  then  we  can  define  a  resampling  of 
data  defined  on  the  C  -grid  to  the  I>-grid  as  follows. 

We  are  given  an  A^-vector  (vq,  •  •  •  ,vv_i)  of  data,  each  Vj  representing  a  value 
on  cell  C  j.   We  define  resampled  data  (wq,  ■  •  •  ,w»y_i)  by 

N-l 


where 


'■y ""  SI^i(^'y^^j(^'y^'^-"'^y 


Here  we  have  in  mind  interpolation  functions  ^j(x,y)  that  are  characteristic  func- 
tions of  the  corresponding  cells  Cj,  and  sampling  functions  that  are  characteristic 
functions  of  the  cells  P,-  normalized  to  have  unit  mass.   That  is, 

y\ijix,y)  =  I  1   if  (x,y)^Cj 
^  0  otherwise  , 
and 

i^i{x,y)  =  I  l/area(r,)    if  (x,y)€Vi 
^  0  otherwise  . 
Area,  of  course,  is  measured  by  integration 

area  (2?,)  =  JJ^dxdy  . 

More  general  interpolation  and  sampling  kernels  can  be  envisioned,  but  the  simplest 
such  kernels,  the  characteristic  function  of  the  cells  as  described  here,  should  prove 
adequate  for  effective  resampling. 

5.2.  Polar  sampling 

We  now  discuss  a  special  sampling,  the  polar  sampling,  which  is  important  for 
snapshot  visual  perception.  In  polar  sampling,  the  sample  points  are  located  on  the 
intersections  between  a  set  of  rays  and  a  set  of  concentric  circles.  Some  models  of 
human  retinal  receptor  distribution  describe  samples  points  in  this  way,  where  reso- 
lution falls  off  with  the  distance  (eccentricity)  from  the  fovea.  Sometimes  linear 
fall-off  of  the  sampling  rate  is  assumed,  which  leads  to  placement  of  grid  points  on 
concentric  circles  whose  radii  increase  as  \og(A+kh),  A:  =  l,2,  •  •  •  ,  where  A  and  h 
are  constants  [9].  Placement  of  the  circles  with  radii  increasing  exponentially  [11] 
and  also  simply  increasing  linearly  [12]  are  also  known  (see  Figure  10). 
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We  can  consider  resampling  a  polar  grid  by  reducing  the  number  of  concentric 
circles,  or  we  can  reduce  the  number  of  rays.  When  the  concentric  circles  are 
reduced,  we  simply  resample  the  grid  points  along  each  ray  independently  of  the 
other  rays.  Resampling  a  ray  is  straightforward  using  the  methods  of  Section  2. 
When  the  number  of  rays  is  reduced,  the  points  on  each  concentric  circle  can  be 
resampled  independently  of  the  other  circles.  To  do  this,  the  notion  of  one- 
dimensional  resampling  of  a  line  segment,  as  described  in  Section  2,  has  to  be 
extended  to  resampling  along  a  circle.  However,  the  methods  of  Section  2  extend 
easily,  using  distance  based  on  radian  measure  in  place  of  one-dimensional 
Euclidean  distance. 

If  both  the  number  of  concentric  circles  and  the  number  of  rays  are  to  be 
reduced,  we  can  then  do  the  resampling  first  by  reducing  the  number  of  circles,  and 
then  by  reducing  the  number  of  rays.  As  with  two-dimensional  resampling,  this 
process  is  unaffected  if  we  exchange  the  order  of  resampling.  Further,  the  optimal- 
ity  properties  of  the  weights  used  for  resampling,  alluded  to  in  Section  1,  extend  to 
polar  resampling.  Indeed,  because  the  polar  grid  coordinate  locations  can  be 
decomposed  into  a  cross  product  of  circles  and  rays,  a  separable  kind  of  adaptive 
resampling  is  possible  for  polar  grid  pyramids,  similar  to  the  adaptive  tv.'o- 
dimensional  rectangular  pyramids  discussed  in  Section  4.2. 

6.  Summary 

Using  the  anti-aliasing  method  common  in  computer  graphics  leads  to  an  idea 
for  one-dimensional  resampling  of  N  points  into  M  points.  Extending  this  idea  to 
two  dimensions  easily  establishes  a  method  for  building  pyramids  with  arbitrary 


Figure  10.  Polar  sampling,  where  sampling  points  are  on  the  intersection  points 
between  lines  and  circles.  The  left  sampling  is  with  exponential  radius  growth,  while 
on  the  right  the  growth  of  the  radius  is  linear. 
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sizes  specified  for  each  level.  We  thus  see  that  the  use  of  pyramids  with  dimensions 
given  by  powers  of  two  is  an  unnecessary  restriction  on  the  construction  of  the 
pyramids.  The  question  as  to  what  size  levels  are  most  appropriate  is  left 
unanswered,  and  depends  upon  the  application  and  empirical  experiences. 

We  have  also  investigated  the  idea  of  adaptive  resampling.  The  idea  is  easy  in 
one  dimension,  and  allows  for  arbitrary  nonlinear  stretching  along  the  line  to  be 
resampled.  We  gave  one  particular  busyness  measure  to  use  as  a  basis  for  deciding 
on  the  stretching.  In  two  dimensions,  things  are  more  complicated,  and  we 
compromised  by  permitting  only  a  "separable  stretching,"  where  the  rows  are 
resampled  using  one  set  of  weights,  yielding  the  same  stretching  for  all  rows,  and 
then  the  columns  are  resampled  using  a  single  set  of  weights. 

Finally,  we  note  that  the  idea  extends  to  irregular  tessellations,  where  sample 
points  can  be  randomly  placed,  or  placed  on  polar  grid  or  hexagonal  patterns. 
Ideally,  adaptive  resampling  would  allow  one  to  dynamically  place  additional  points 
in  regions  with  high  busyness,  but  our  method  does  not  easily  extend  to  the  case  of 
adaptive  placement  of  resample  point  locations  in  two  dimensions. 
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